Tightening Pod/Container Security for All Workloads by alix-graylog · Pull Request #100 · Graylog2/graylog-helm

alix-graylog · 2026-06-26T20:11:58Z

Summary

Locking down default pod and container security contexts, as far as the application currently allows, and exposing the configuration to the values.yaml

Details

Updated Graylog application and it's init container's security context's and moving them into user configurable values
Updated Graylog datanode ~~and it's init container~~'s security context's and moving them into user configurable values
Updated geoips security contexts

What this does

Moves security contexts out of hardcoded template values and into values.yaml so they're configurable, and tightens the defaults for both Graylog and Datanode.

podSecurityContext and containerSecurityContext are now values-driven for both workloads
Graylog init containers get their own initContainerSecurityContext — they don't need NET_BIND_SERVICE since they never start the JVM
copy-plugin-* init containers get readOnlyRootFilesystem: true (they only write to an emptyDir)
Datanode gets a data-chown init container (busybox) that handles ownership setup on the PVC before the main container starts. The image is configurable via datanode.initContainerImage for private registry support
seccompProfile: RuntimeDefault applied across all containers
allowPrivilegeEscalation: false and capabilities: drop: ALL on everything

Why some things couldn't be locked down further

Graylog — NET_BIND_SERVICE has to stay. The JDK binary in the image has cap_net_bind_service=ep set as a file capability. With no_new_privs=1 (which allowPrivilegeEscalation: false sets), executing a binary that would gain capabilities not already in the process's permitted set fails with EPERM. Pre-adding NET_BIND_SERVICE to the container's permitted set means the file capability doesn't grant anything new, so the exec works.

Datanode — has to start as root. This is common with Opensearch in Kubernetes. The entrypoint does chown -R graylog:graylog /var/lib/graylog-datanode then exec setpriv --reuid=graylog --regid=graylog --init-groups unconditionally. That requires CHOWN, DAC_OVERRIDE, SETUID, and SETGID. We tried moving the chown to the init container to strip those from the main container, but the entrypoint still runs its own chown -R on startup — and root without CAP_CHOWN can't chown files owned by uid 999. We also tried setting runAsUser: 999 directly, but the container runtime clears the effective capability set when it switches from root to non-root, so SETGID ends up in the bounding set but not effective — setpriv --init-groups still fails. After setpriv runs the process has uid 999 and no capabilities, so the running application is fine, just the startup window needs them.

readOnlyRootFilesystem — not set on the main containers. Both the Graylog JVM and the OpenSearch-based datanode write to the container filesystem at runtime and would need a full audit + emptyDir mounts to enable this safely.

Opting out

If the security contexts cause issues in your environment, each field is fully overridable. To disable them entirely, set the value to null — note that {} does not work: Helm coalesces an empty map back to the chart defaults, so the default context still renders. null is required to actually clear the field.

graylog:
  podSecurityContext: null
  initContainerSecurityContext: null
  containerSecurityContext: null

datanode:
  podSecurityContext: null
  containerSecurityContext: null

Linked issues

improve securityContexts #70

PR Checklist

Please check the items that apply to your change.

Testing Checklist

Static Validation

Linter check passes: helm lint ./charts/graylog
Helm renders local template sucessfully: helm template graylog ./charts/graylog --validate

Installation

Fresh installation completes successfully: helm install graylog ./charts/graylog
All pods reach Running state: kubectl rollout status statefulset/graylog
Helm tests pass: helm test graylog

Functional (if applicable)

Web UI accessible and login works
DataNodes visible in System > Cluster Configuration
Inputs can be created and receive data

Upgrade (if applicable)

Upgrade from previous release succeeds
Scaling up/down works correctly
Configuration changes apply correctly

Specific to this PR

Datanode workloads started successfully.
Graylog application workloads started successfully
GeoIP workloads started successfully

Notes for reviewers

Verify all applicable tests above pass
Validate that the linked issues are no longer reproducible, if applicable
Sync up with the author before merging
The commit history should be preserved - use rebase-merge or standard merge options when applicable

…ylog and geo-ip

Copilot

Pull request overview

This PR tightens the default Pod/Container security contexts for Graylog, DataNode, and GeoIP update jobs, and makes those security settings configurable via values.yaml so operators can adjust or opt out as needed.

Changes:

Added values-driven podSecurityContext, initContainerSecurityContext, and containerSecurityContext defaults for Graylog and DataNode.
Updated StatefulSet templates to render the new values-driven security contexts, and added a DataNode data-chown init container.
Updated GeoIP CronJob/Job rendering to accept Pod/Container security contexts via the shared helper.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
charts/graylog/values.yaml	Introduces configurable default security contexts (and initContainer image config) for Graylog and DataNode.
charts/graylog/templates/workload/statefulsets/graylog.yaml	Renders new values-driven pod/container/init-container security contexts for Graylog workload and plugin-copy init containers.
charts/graylog/templates/workload/statefulsets/datanode.yaml	Adds values-driven pod/container/init-container security contexts plus a `data-chown` init container for PVC preparation.
charts/graylog/templates/workload/cronjobs/geoip.yaml	Passes security contexts into the GeoIP JobSpec helper.
charts/graylog/templates/_helpers.tpl	Extends GeoIP JobSpec helper to optionally render pod/container security contexts.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

… deployment

Since the datanode container currently must run with root privileges, the data-chown init container becomes redundant and adds complexity. The datanode entrypoint unconditionally creates and chowns the data directories as root before dropping privileges, so the busybox data-chown init container only duplicated that work without letting the main container shed any capabilities. Removing it also drops the busybox:latest public-registry pull. Also removes the now-unused datanode.initContainerImage and datanode.initContainerSecurityContext values. The main container's security context is unchanged. Ref: https://github.com/Graylog2/graylog-docker/blob/937a318/docker/datanode/entrypoint.sh Co-Authored-By: Claude Opus 4.8 (1M context) <[EMAIL_ADDRESS_REDACTED]>

…ainers add a conditional rendering for the security context of `copy-plugin-` init containers based on `.Values.graylog.initContainerSecurityContext`: if `initContainerSecurityContext: {}` no security context is merged, otherwise the intended merge takes place Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Guard the pod/init/container security contexts added for the graylog and datanode workloads and the geoip update CronJob (issue #70): - graylog StatefulSet: default pod/init/container contexts, copy-data init container has no NET_BIND_SERVICE, graylog-app keeps it, copy-plugin init containers are forced readOnlyRootFilesystem, and the opt-out path. - datanode StatefulSet: new suite covering the pod context and the startup capabilities on the main container, plus opt-out. - geoip CronJob: the graylog pod/container contexts thread through the graylog.geoip.job.spec helper's positional args, plus opt-out. Opt-out is asserted with `null`, not `{}`: an empty map is coalesced back to the chart defaults by Helm, so only null actually clears the field. Co-Authored-By: Claude Opus 4.8 (1M context) <[EMAIL_ADDRESS_REDACTED]>

Add podSecurityContext / initContainerSecurityContext / containerSecurityContext to the graylog block and podSecurityContext / containerSecurityContext to the datanode block, matching the values.yaml added for issue #70. Typed as ["object", "null"] so the null opt-out validates — an empty map ({}) is coalesced back to the chart defaults by Helm, so null is the only value that actually clears the field. Co-Authored-By: Claude Opus 4.8 (1M context)[EMAIL_ADDRESS_REDACTED]om>

The copy-plugin securityContext used merge (dict "readOnlyRootFilesystem" true) over the user's initContainerSecurityContext. Because mergo treats false as a zero-value, a user setting readOnlyRootFilesystem: false could never override the true default. Default it with hasKey instead: only inject readOnlyRootFilesystem: true when the user hasn't set the key, so an explicit false is preserved. This also lets the block use `with` on the raw value, which self-skips on the null opt-out, dropping the separate if-guard. Co-Authored-By: Claude Opus 4.8 (1M context) <[EMAIL_ADDRESS_REDACTED]>

monrax · 2026-07-09T22:35:22Z

Made a few changes:

dropped the datanode data-chown init container: since DN still needs to be run as root, its init container becomes redundant, especially considering the container entrypoint already does the exact thing the init container intended to do.
added unittests
added security context values to values.schema.json
fixed the merge issue for copy-plugin: the original merge (dict "readOnlyRootFilesystem" true) didn't allow for readOnlyRootFilesystem: false overrides due to merge treating that as an empty value, and defaulting to the hard-coded true one instead. This also didn't leave space for opting out. Both are fixed by just injecting the field conditionally inline before the toYaml

Add a Hardened Environments section to the README covering the PSA exemptions the datanode and MongoDB operator currently require (or BYO MongoDB), and clarify the fsGroup comment in values.yaml. Allow maxmindGeoIp.accountId to be an integer in values.schema.json (the template already casts it), so a bare --set works without --set-string. Warn in NOTES.txt when graylog.podSecurityContext.fsGroup is unset, since Graylog runs non-root and needs it to write its volume on storage that enforces ownership (e.g. EBS). Co-Authored-By: Claude Opus 4.8 (1M context) <[EMAIL_ADDRESS_REDACTED]>

monrax · 2026-07-10T06:39:09Z

Did a validation run on EKS:

Environment

EKS
3× m6i.2xlarge
Amazon Linux 2023
k8s 1.35
containerd 2.1.5
kernel 6.12
EBS CSI
MongoDB MCK operator

Test matrix

Check	Result
Core — fresh install, full default footprint	✅ pass
Upgrade — pre-hardening (main) → this branch	✅ pass (volume ownership reconciled)
Neg-Graylog — strip NET_BIND_SERVICE	✅ fails as designed
Neg-Datanode — strip startup caps	✅ fails as designed
Opt-out — null all contexts	⚠️ breaks Graylog on EBS — see below
Override — custom fsGroup	✅ pass (deep-merges, applies at runtime)
Restart — delete stateful pods	✅ pass (fsGroup idempotent on re-attach)
Scale up/down — 1 ↔ 2 replicas	✅ pass (new replicas hardened + writable)
copy-plugin — readOnlyRootFilesystem: true init container	✅ pass
GeoIP — MaxMind download under new secctx	✅ pass
PSA — enforce=restricted namespace	✅ characterized (graylog admits, datanode needs baseline)

Confirmed claims

Graylog NET_BIND_SERVICE is load-bearing. Removing it (capabilities.add=[]) crashes the JVM exec exactly as the PR predicts:

/docker-entrypoint.sh: line 129: /opt/java/openjdk/bin/java: Operation not permitted

With the cap present, Graylog runs clean as uid=1100, effective caps 0 at runtime (cap only needed for the file-cap exec transition).

Datanode needs its five startup caps. Dropping them fails the entrypoint chown:

install: cannot change owner and permissions of '/var/lib/graylog-datanode/opensearch/config': Operation not permitted

With the caps, the container starts as root (PID 1) and the workload drops to uid 999: OpenSearch (opensearch-2.19.3) and the datanode JVM both run as 999; effective caps 0xcb (i.e. exactly CHOWN,DAC_OVERRIDE,FOWNER,SETGID,SETUID, nothing extra). NoNewPrivs=1 on every container.

Upgrade reconciles existing volumes. From a pre-hardening main install (datanode volume 999:999 755, no fsGroup), helm upgrade to this branch triggered kubelet's fsGroupChangePolicy: OnRootMismatch to fix the mode → 999:999 2775, data writable, 0 crashloops.
copy-plugin RO rootfs works. The init container renders readOnlyRootFilesystem: true and still completes exitCode 0; the jar lands in the plugin dir.
GeoIP works under the new context. Post-install hook Job downloaded both DBs to the shared PVC as fsGroup 1100:

-rw-r--r--. 1 graylog graylog 65812200 GeoLite2-City.mmdb
-rw-r--r--. 1 graylog graylog 12134013 GeoLite2-ASN.mmdb

The geoip container inherits graylog.containerSecurityContext (incl. NET_BIND_SERVICE + no_new_privs); the geoipupdate binary has no file caps, so it execs fine.

One minor "regression"

The opt-out example sets graylog.podSecurityContext: null, which removes fsGroup: 1100. On EBS a fresh PVC mounts root:root, and Graylog runs non-root with no root-chown step in its entrypoint (unlike the datanode), so it can't create its data dirs:

init copy-data: cp: cannot create directory '/mnt/data/config': Permission denied
graylog-app:    mkdir: cannot create directory '/usr/share/graylog/data/config': Permission denied

This is a new failure mode: the pre-hardening chart hardcoded fsGroup:1100, so it never happened before. The datanode survives opt-out (root + self-chown); Graylog does not.

Solution: since nulling out could work on valid setups that genuinely don't need fsGroup (e.g. storage that ignores ownership), instead of injecting the non-root fsGroup or forbidding it in values.schema.json, we just add both a comment to values.yaml and a non-fatal warning to NOTES.txt

Good to merge!

monrax

works

feat(security-context): Hardening security contexts for datanode, gra…

570ae11

…ylog and geo-ip

alix-graylog marked this pull request as ready for review June 26, 2026 20:37

alix-graylog requested review from Copilot, monrax and williamtrelawny June 26, 2026 20:37

Copilot started reviewing on behalf of alix-graylog June 26, 2026 20:38 View session

Copilot AI reviewed Jun 26, 2026

View reviewed changes

Comment thread charts/graylog/templates/workload/statefulsets/graylog.yaml Outdated

Comment thread charts/graylog/templates/workload/statefulsets/datanode.yaml Outdated

Comment thread charts/graylog/values.yaml Outdated

Comment thread charts/graylog/values.yaml

fix(security-context): Adding SETGID to datanode to allow for initial…

8076d80

… deployment

alix-graylog changed the title ~~Tightening Pod/Container Security for All Workloada~~ Tightening Pod/Container Security for All Workloads Jul 1, 2026

monrax mentioned this pull request Jul 9, 2026

Add conditional root check to datanode entrypoint Graylog2/graylog-docker#306

Open

6 tasks

monrax and others added 6 commits July 9, 2026 23:08

Merge branch 'main' into security-contexts-2

8015851

monrax added 2 commits July 10, 2026 07:50

update usage comment on geoip helper

0c23e1b

monrax approved these changes Jul 10, 2026

View reviewed changes

monrax merged commit cb9bd63 into main Jul 10, 2026
9 checks passed

monrax deleted the security-contexts-2 branch July 10, 2026 06:48

monrax mentioned this pull request Jul 10, 2026

Security Hardening: run DataNode fully non-root #109

Open

25 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Tightening Pod/Container Security for All Workloads#100

Tightening Pod/Container Security for All Workloads#100
monrax merged 10 commits into
mainfrom
security-contexts-2

alix-graylog commented Jun 26, 2026 •

edited by monrax

Loading

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

monrax commented Jul 9, 2026

Uh oh!

monrax commented Jul 10, 2026

Uh oh!

monrax left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

alix-graylog commented Jun 26, 2026 • edited by monrax Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

What this does

Why some things couldn't be locked down further

Opting out

Linked issues

PR Checklist

Testing Checklist

Static Validation

Installation

Functional (if applicable)

Upgrade (if applicable)

Specific to this PR

Notes for reviewers

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

monrax commented Jul 9, 2026

Uh oh!

monrax commented Jul 10, 2026

Environment

Test matrix

Confirmed claims

One minor "regression"

Uh oh!

monrax left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

alix-graylog commented Jun 26, 2026 •

edited by monrax

Loading